Inject Havoc

To begin conducting chaos experiments, users are to access the Inject Havoc module. With extensive chaos experiment capabilities spanning infrastructure, network & application layers, one can easily identify areas for improving the resiliency of mission critical applications.

On the NetHavoc window, click Inject Havoc menu item on the top bar as shown in the screenshot below. In this section, a user can configure and apply havoc that needs to be injected.

The havoc to be configured are divided into four categories. Each category has several types. The user needs to choose the category and its type and proceed to configure the havoc. These categories, their types and other functionalities are described below.

Platform Supported

Linux
Windows
VMWare, TAS
Kubernetes
Multiple cloud environments (AWS, Azure, GCP)

Common Fields

The following fields are commonly observed across all platforms and are used to define havoc properties like the target, duration and havoc injection time:

Injection Time

Injection Time signifies the time at which the faults are to be injected based on the configurations. You have the following options for fault injection time:

Current: Here, the system captures the current time and injects havoc for the specified duration from the current date and time.
Specified: Here, you need to specify the exact date, time, and duration (in HH:MM: SS format) at which the configured fault is to be injected.
Random: You can specify the start date and time and end date and time along with the duration (in HH:MM: SS format) for injecting faults. The system picks any random time between the specified time and injects faults based on the provided duration.
Duration:This section displays the total duration of the injected havoc. The maximum duration is 99hr:59m:59s.
Number of Iterations:It is the number of iterations to be performed over a given message set that can have one or more messages.

· If any havoc is already at Running or Scheduled status, another havoc with Injection Time as Specified or Random cannot be configured. It can be configured if the Injection Time is Current.

· If a user opens a test run from the UI, and then opens NetHavoc, it opens with the test run at that time.

Duration

Duration displays information about the total duration of the injected havoc. This information is divided into three stages – Shoot Up, Stable, and Tumble. This information is also displayed in the form of a graph.

Shoot Up: This is the ramp-up phase.
Stable: This is the duration phase.
Tumble: This is the ramp-down phase.

This is disabled when the selected Havoc Type is Application Termination or Server Termination.

Server Info

In Server info, the user has to select the various server information as per their requirement. A user needs to specify tier, server selection mode, server, and connection from the corresponding drop-down lists.

The user has to fill in the following fields from the respected drop-down:

Tier: The tier on which havoc is to be injected.
Server Selection Mode: Choose if you want to select a specified or a dynamic server.
Specified: Select the specific server(s).
Dynamic: Select any number or any percentage of available servers.
Server: Based on the specified or dynamic server selected earlier, select the server on which havoc is applied.
ALL (Dynamic server): If you select this option, all the servers are selected.
ANY (Dynamic server): Provide any number of servers.
ANY % (Dynamic server): Provide any percentage of available servers.
Connection Type: Havocs are injected on the server through these connection types (modes) – SSH and Cavisson Agent.

Category and Havoc Types

NetHavoc’s extensive chaos experiments or havocs are broadly divided into the following categories. This section explains each of these categories along with the various configuration options under each.

The different havoc categories are:

Starve Application
State Change
Network Assaults
Application Disruption

Starve Application

As the name suggests, starve application havoc directly impacts the resources of the machine on which the application resides to check how quickly your system recover in case of unexpected outages causing excessive resource consumption. The havoc types for this category are as follows:

CPU Burst

The CPU Burst havoc provides you the option to consume the CPU of the server on which the application is deployed. You can either increase the CPU utilization or fill the CPU cores to check how your application behaves when CPU related resources are excessively consumed or completely exhausted.

CPU Attack: Through CPU burst user can inject havoc into server. In this havoc Exhaust/consume CPU resources via two different methods:
Fill Core: The number of cores in the CPU that you want to consume. You can fill N number of cores, where N is the total number of cores available in the system.
- CPU Cores: Enter the number of cores that you want to fill via this havoc. Note that you can see the available cores for the selected server right next to this input box.
CPU Utilization %: Consumes CPU on mentioned server(s). Via this havoc, you have the option to increase the CPU utilization percentage on all cores.
- CPU Burn (When ‘CPU Utilization %’ is selected as CPU Attack): Enter the value by which you want to increase the CPU utilization.

Note: CPU burn will only show in case the user has selected the CPU Utilization% option for the CPU Attack.

Disk Swindle

Through the disk swindle havoc, users can fill up the disk of the selected server by filling up disk partitions and subsequently analyze the impact on the infrastructure and application.

The user has to fill in the following fields:

Partition Name: The user has to select the partition on which they want to inject the havoc. Once the user clicks on the partition name field, the Partition Name window will appear as a pop-up. The user has to select the partition they want to use. Once selected click on the Add button to add the partition.
Partition Value: Provide the % value defining the amount of disk to be filled for the selected partition.

I/O Shoot Up

I/O Shoot Up impacts the file system by increasing the input/output (I/O) activity on the selected instance/nodes. Provide information for the following fields to inject this havoc:

Operation: Select the I/O operation to be performed on the selected node. The user can select any of the following:
Read – Perform read operations on the node.
Write – Perform write operations on the node.
Read/Write – Perform both read and write operations on the node.
Workers: Provide the number of workers that will run in parallel to execute the I/O Shoot Up havoc.
Block Size: Provide the block size of the file that will be used to perform the read or write operation. The value provided will be in KB.
Block Count: Provide the count of blocks or files which are to be used for the selected operation i.e. if you provide the block size as 10 KB and count as 4 then four files of 10KB each will be used to inject the havoc.

Memory Outlay

This havoc consumes system memory by increasing RAM utilization to help you assess how your application behaves when your system’s RAM is on the verge of exhaustion or has already been exhausted.

The user has to fill in the following fields which are listed below:

Memory Unit: In this field, you can select the unit for memory consumption i.e. you can either consume RAM by specifying the value in MB, GB or Percentage.
Target Memory: Provide the numerical value of the memory to be consumed according to the unit selected. The available memory of the selected server is shown adjacent to this input box.

State Change

The state change havoc category alters the application or machine’s state via multiple options. These types of havoc are instrumental in bringing about unexpected changes to the system or application by conducting chaos experiments on the machine and application levels.

Abort Application

Through the abort application, the user can inject havoc on any process or application of the server through the process name and process ID.

- Process Category: Gives you the option to select either the process ID or process name or PID file via which the application process will be terminated.
- Process Entry: Provide the corresponding process ID or name depending upon the values selected under process category.
Terminates Cloud Instance
In Terminates Cloud Instance havoc type, you can terminate an instance on Azure, AWS, or GCP. Depending upon the cloud service provider you select, you will have to provide the relevant details accordingly, as described below:
Cloud Type: The cloud server on which the instance is to be terminated (Azure or AWS).
When you select ‘Azure’ as the cloud type:

User Name/Password: Credentials via which the cloud instance will be accessed.
Tenant ID: Provide the unique identifier of the Azure Active Directory instance.
Resource Group Name: They are logical containers for a collection of resources that can be treated as one logical instance. Users can use resource groups to control all of their members collectively.
Computer Name: Provide the instance IP or name which will be used to terminate the specified instance according to the tenant ID mentioned above.

When you select ‘AWS’ as the cloud type:

- Region Name: Select the region where your AWS instance is located.
- Access ID: Provide the unique access ID that will be used to terminate the instance.
- Secret Access Key: Provide the AWS secret access key for accessing the instance.
- Instance ID: Provide the unique instance ID of the instance that you want to terminate.
When you select ‘GCP’ as the cloud type:

Google Service Account: Add the account name & its corresponding service account key by clicking on the icon. This information is used to generate the list of projects and instances.
Project List: Select the project from the project list drop-down.
Instance Name: Select the instance name from the drop down that will be terminated.

Note: If Google Service Account is already configured, users can directly select from the drop down.

Kill Server

Perform a shut down or reboot operation on the selected server to check the failover capabilities of your system.

Once the target server is selected, you can perform the following operations:

Operation: The type of operation in which you want to impact the machine, such as shut down or reboot.
Shut Down: Selecting this mode kills the machine.
Reboot: Selecting this mode reboots the machine.
Delay: The havoc will be executed after the specified delay time (in minutes).

Teleport

The teleport havoc changes the system time to check how the application reacts to such a change.

You can do that in the following two ways:

Travel to: Select whether you want to change the system time in the past or the future from the current time.
Future: You can change the system time to future with respect to the defined duration.
Past: You can change the system time to past with respect to the defined duration.
Duration: Duration from the current time (in HH:MM: SS format).

Distort JMS Queue

Distort JMS Queue havoc is used for sending lots of messages to check the resiliency of the JMS server or sending incorrect messages for checking application behavior. In this, the JMS type is selected from the drop-down where you want to inject the havoc. JMS types are mainly of three types:

IBMMQ
Kafka
TIBCO

Configuring IBMMQ: You need to provide the following details within the JMS All fields are mandatory.

Server Name/ IP: This is the IBM MQ Server Name/ IP.
Port: This is the port where the IBM MQ is listening.
Queue Manager: This manages the resources associated with a queue that it owns.
Channel: This is used to transfer messages between queues.
Queue: This is responsible to hold the message until the receiver is ready.

For details refer to the links- Introduction to IBM WebSphere MQ – IBM Documentation

Configuring Kafka: You need to provide the following details within the JMS Configuration for Kafka:

Server Name/IP: This is the Kafka Server IP/Hostname. It is a mandatory field.
Port: This is the port where Kafka is listening. It is a mandatory field.
Topic: This is used as a message-oriented middleware that is responsible to hold and deliver messages. It is a mandatory field.

Additional Settings for Kafka: To secure the connection, use SSL settings. You can configure the following:

SASL (Simple Authentication Security Layer) Settings:
- User: Enter the SASL username.
- Password: Enter the SASL password.
- Mechanism: The SASL supports various mechanisms for authentication, such as “PLAIN “, “non-plaintext “, etc. Select a mechanism as per the requirement.
Other Settings: When you select the Enable SASL Authentication checkbox, the system displays the following fields:
- Message Key: The Message Key is a message identifier that helps to distinguish the clients as each client has a unique key.
- Header Key: Fill in the header key in this field.
- Header Value: Fill in the header value in this field.

You have to fill in the following fields in the SSL Settings:

Use Specific Ciphers:From the dropdown, select the Cipher needed.
Key File: Key files are open SSL generated keys with the crypto toolkit and saved into files with the. key or .pem extension. There is also a password for. key file.
Trusted CA file: These are the trusted certificates, which are data files, used to cryptographically link an entity with a public key.

Once you click the Kafka is enforcing Client Authentication, you get another option/field Certificate File in SSL Settings which are small data files that digitally bind a cryptographic key to an organization’s detail.

For more details refer to the link- Apache Kafka

Configuring TIBCO: You need to provide the following details within the JMS Configuration for TIBCO:

Server Name IP: This is the TIBCO server IP/hostname. It is a mandatory field.
Port: This is the port where TIBCO is listening. It is a mandatory field.
Topic/Queue: The queue is responsible to hold the message until the receiver is ready. The topic is used as a message-oriented middleware that is responsible to hold and deliver messages. The user can either select Topic or Queue.

Additional Settings for TIBCO: In Additional Settings, you get the Other Settings option, after clicking the Other Security, you get the following fields:

Message Key: The Message Key is a message identifier that helps to distinguish the clients as each client has a unique key.
Header Value: Fill in the header value in this field. Using SSL Settings, you have to fill in the following fields in the SSL Settings:
Using SSL Settings: You have to fill in the following fields in the SSL Settings:
Use Specific Ciphers: From the dropdown, select the Cipher needed.
Trusted CA file: These are the trusted certificates, which are data files, used to cryptographically link an entity with a public key.

Once you click the Kafka is enforcing Client Authentication, you get another options/fields Certificate File and Key File in SSL Settings

Certificate File and Key File in SSL Settings: Certificate File Field- which are small data files that digitally bind a cryptographic key to an organization’s detail.
Key File: Key files are open SSL generated keys with the crypto toolkit and saved into files with the. key or. pem extension. There is also a password for .key file.

For more details refer to the link- JMS Message Models (tibco.com)

Network Assaults

The network assault category of havoc allows you to conduct chaos experiments on the network layer to test your system’s and application’s resilience. The havoc introduces network corruption, latency, packet loss, and others in this category. The havoc types for this category are as follows:

Intrude Network

A user can corrupt incoming or outgoing network packets to and from the server(s). The default value is 10% and the maximum can be 50% corruption.

A.) Network Interface: It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that have been added to the server.

b.) Corruption Percentage: The % of network packets that need to be corrupted or manipulated.

Trim Network Packets

You can apply loss in the incoming and outgoing packets to the server(s) on the user-defined interface. The default value is 10% and the maximum can be 50% loss.

Network Interface: Provide the network interface. The drop-down lists all the interfaces that have been detected from the server that you select under the Server Info section.
Loss: The % of packets lost in the network.

Dormant Network

Using this havoc type, you can generate a delay in the packet receiving and sending to the server(s). You can provide a fixed value or a range (in milliseconds).

- Network Interface: It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that have been added to the server.
- Mode: Select the mode whether you want to delay the network traffic by a specified amount of time or between a range of time.
- Specified: Provide the integer value of the field. The packets will be delayed by the specified amount of time.
- Random: The packets will be delayed by any time between the two given values.
When the mode is ‘Specified’:
- Delay Value: The specified amount of time (in milliseconds) by which network traffic is to be delayed.
When the mode is ‘Random’:
- First Delay Value: The first value of the range of time (in milliseconds) by which network traffic is to be delayed.
- Second Delay Value: The second (end) value of the range of time (in milliseconds) by which network traffic is to be delayed.
DNS Breakdown
With the help of this havoc type, you can reject the calls on the DNS server on a specified or all interfaces. You can block the traffic coming from a single IP or a range of IPs. You can also block the traffic based on either TCP or UDP protocol.

The user has to fill in the following fields from the respected drop-down:

Network Interface: It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that have been added to the server.
DNS Server Selection: Select the mode for which you want to reject the calls on the DNS server (such as Single, Range, or IP with CIDR).

If the host is selected as ‘IP range’, there are the following fields:

From DNS IP: The IP address for which you want to impact the network traffic.
To DNS IP: The IP address for which you want to impact the network traffic.

If the host is selected as IP Subnet, it has the following fields:

DNS IP: The IP address for which you want to impact the network traffic.
Subnet Bits: The Classless Inter-Domain Routing(CIDR) value of IP address.
Protocol: The protocol that you want to be impacted, such as TCP or UDP.

Interstellar

You can choose this havoc type to drop all ingress/egress matching traffic on specified or all interfaces. You can block the ingress/egress traffic based on a single IP or a range of IPs. You can also block the ingress/egress traffic based on TCP/UDP/ICMP protocol.

The user has to fill in the following fields which are listed below from the respected drop-down:

Network Interface: It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that have been added to the server.
Server Mode: Select the mode for which you want to reject the calls on the DNS server (such as Single, Range, or IP with CIDR).
- Single: Select this when you have a single server.
- IP Range: In this field, select the IP range by entering both the DNS IP.
- IP Subnet: In IP Subnet, select a DNS IP and select the Subnet bits.
Destination Port: Add a port number of the destination.
Direction: Select the traffic mode from the drop-down from either egress or ingress. Egress implies the traffic that exits the network and ingress signifies the traffic that enters the network.
Protocol: Click on the drop-down to select a protocol – TCP or UDP.

Application Disruptions

This category of havoc allows you to conduct chaos experiments on the application directly and check the resiliency of your critical applications. Multiple types of havoc are supported to inject faults across methods and/or services.

The havoc types for Application Disruptions are as follows:

Alter Inbound Services

Delay response time of services or cause service failure in service transactions to check how efficiently your application handles issues with critical services/transactions and maintain business continuity.

- BT Name: From the drop-down select the option to provide the name of the business transaction of the target request. By default, the ALL option is selected.
  - ALL: Select all the available business transactions.
  - Specified Business Transaction(s): Select the specific name of business transaction(s) from the dropdown.
  - Business Transaction Name: Enter the name of the business transaction.
  - Request URL: Enter the relative URL of the target request.
- Operation: Select the type of operation in which you want to Alter the Inbound Services either Delay in Response Time or Service Failure.
- Delay (When Delay in Response Time is selected as Operation): Enter the specified amount of time (in milliseconds) by which service failure is to be delayed. The default value is 500 ms.
- Apply on _____%Transactions: Specify the desired percentage of transactions to be impacted. The default percentage is 100%.
Alter Outbound Services
Alter Outbound Service induces delayed response time or service failure between a node and an integration point to check the resiliency in face of outages and unexpected issues.

BT Name: From the drop-down select the option to provide the name of the business transaction of the target request. By default, ALL option is selected.
- ALL: Select all the available business transactions.
- Specified Business Transaction(s): Select the specific name of business transaction(s) from the dropdown.
- Business Transaction Name: Enter the name of the business transaction.
- Request URL: Enter the relative URL of the target request.
Integration Point: From the drop-down select the option to provide an Integration point. By default, ALL option is selected.
- All: Select all the available integration points.
- Specified Integration Points: Select the specific of integration point(s) from the dropdown.
- Integration Point Name: Enter the name of the integration point.
- Destination Host: Enter the destination host for integration point call out.
- Destination IP Addresses: Enter the IP Address.
Operation: Select the type of operation in which you want to Alter the Outbound Services, such as Delay in Response Time or Service Failure.
Delay (When Delay in Response Time is selected as Operation): Enter the specified amount of time (in milliseconds) by which service failure is to be delayed. The default value is 500 ms.
Apply on ____ % Integration point calls: Specify the desired percentage of integration point calls to be impacted. The default percentage is 100%.

Method Invocation

Method invocation refers to when we invoke havoc on application methods that have method monitors in place. Method Invocation Fault helps you simulate a condition where calls to Java methods can be delayed by a specific time.

BT Name: From the drop-down select the option to provide the name of the business transaction of the target request. By default, ALL option is selected.
- ALL: Select all the available business transactions.
- Specified Business Transaction(s): Select the specific name of business transaction(s) from the dropdown.
- Business Transaction Name: Enter the name of the business transaction.
- Request URL: Enter the relative URL of the target request.
Method selection:
- Specified Method: Select the specific method name(s) from the dropdown.
- Method Name: Enter the fully qualified name of the java method.
Delay: Enter the specified amount of time (in milliseconds) by which the method call is to be delayed.
Apply on ____%method executions: Specify the desired percentage of method executions to be impacted. The default percentage is 100%.

Method Exception

In Method Exception, we inject exceptions in methods which are being captured by the method monitors.

- BT Name: From the drop-down select the option to provide the name of the business transaction of the target request. By default, ALL option is selected.
  - ALL: Select all the available business transactions.
  - Specified Business Transaction(s): Select the specific name of business transaction(s) from the dropdown.
  - Business Transaction Name: Enter the name of the business transaction.
  - Request URL: Enter the relative URL of the target request.
- Method Selection:
  - Specified Method: Select the specific method name(s) from the dropdown.
  - Method Name: Enter the name of the java method.
- Apply on __________ % method execution: Specify the desired percentage of method executions to be impacted. The default percentage is 100%.
Heap Memory Leak
The heap is the portion of memory where dynamically allocated memory resides. Memory allocated from the heap will remain allocated until the memory is free or the program terminates. The increase in heap utilization in JVM creates a heap memory leak. We support heap memory leaks in the JAVA agent.

Leak Memory ______MB in every _______ms: Enter the value of the heap memory to be leaked in MB and the time in which the leak will occur every millisecond.

Application CPU Burst

We support heap memory leaks in the JAVA agent.

- CPU Attack: It is the way the user wants to consume the CPU resources.
- CPU Burn: Shows the CPU % to be burnt.

Configure Containers as Target

Tier: Select the tier name where the required server is running.
Server: Specify the CMON server (Cavisson Agent) where the required containers are uploaded.
Containers name: Specify the containers where you want to inject havoc. You can select multiple containers running on the selected server.

Configure Kubernetes Nodes/Pods as Targets

It has the following fields:

Cluster: Firstly, configure the cluster from Monitor. After configuring the cluster, select the cluster containing the pods from the dropdown where your target containers are running. A cluster is a set of nodes that runs the containerized applications in Kubernetes.
Namespace: According to the cluster selected, Namespace is filtered, and you can select the Namespace to filter the visible set of pods to inject. \n Using the drop-down menus accordingly, select one or more of your Deployments, StatefulSets, ReplicaSets, DaemonSets, or Pods where you want to cause havoc.
Pod Lists: Preview the targeted pods on which havoc will be injected on the right side of Kubernetes Info.
- Specified: Select a specified Pod to inject havoc.
- ALL: Select all eligible Pods to inject havoc.
- ANY: Select a specified percentage of Pods from any eligible Pods available to inject havoc.
- PATTERN: Select a specified pattern-based Pods from any eligible Pods available to inject havoc.
- Container List: Select Container(s) which you want to target.

Configured Havoc

After doing all the configurations, if the user clicks the Configure & Apply Havoc button, the Havoc Summary window is displayed. But clicking Ok applies the havoc and the user is redirected to the Reports window where the various details of the havoc can be viewed in the Havoc Summary section.

If a user applies havoc on multiple servers, and then clicks Configure Havoc or Configure & Apply Havoc, a havoc summary table displays a list of all those servers whose CMON is not running along with the following two options:

Proceed with remaining Servers
Abort Havoc

If the user selects the ‘Proceed with remaining Servers’ option, UI traverses to the ‘Report’ section and another havoc summary table pops up listing servers that are successfully injected with havoc. When the ‘Abort Havoc’ option is selected, the whole havoc is terminated.

The user can also click Reset to change the configurations of the havoc before applying.

While selecting the server in High CPU Load / Fill Up Disk / Network Corruption havoc type, if CMON isn’t running, an error message is displayed for the user, such as “CMON is not running on Server xx.xx.xx.xx”.

New servers can be upscaled into the running tests of NetHavoc. Users can apply havoc to the upscaled servers both specifically and dynamically. These servers can also be upscaled to the havoc that is scheduled in running tests dynamically.

Apply Havoc

On the NetHavoc window, click Apply Havoc icon as shown in the figure below. In this section, a user can apply all the configured havocs that need to be injected for the specified start time and duration.

The user gets detailed information about the configured havocs, such as category, havoc type, time mode, start time, duration, connection, tier, server, user, havoc details, status, and output. All these havocs are at ‘Ready To Apply’ status.

Click the check box against the havoc that needs to be applied, and then click the Apply Havoc button at the bottom of the window. This displays the Havoc Summary window confirming the activation of the havoc.

Update Havoc

On the NetHavoc window, click the Update Havoc menu icon as shown in the figure below. In this section, a user can edit/update all the havoc applied in the system.

The user gets detailed information about the applied havoc, such as category, havoc type, time mode, start time, duration, status, and output. The havoc can be updated only in case the status is Ready to Apply or Scheduled. If it reaches Running status, the havoc cannot be updated.

Select the havoc which is needed to be updated and click the update icon. This displays a window where the user can change the configurations of that havoc.

After updating the configurations, click the Update Havoc button at the top of the window to apply the changes.

Stop Havoc

On the NetHavoc window, click the Stop Havoc icon as shown in the figure below. In this section, a user can stop the injected havoc forcefully when it is in Scheduled status.

The user gets detailed information about the configured havoc, such as category, havoc type, time mode, start time, duration, and output.

Click the check box against the havoc that needs to be stopped, and then click the Stop Havoc button at the bottom of the window. This stops the injected havoc.

Delete Havoc

On the NetHavoc window, click Delete Havoc as shown in the figure below. In this section, a user can delete the injected havoc from the system. The havoc can be deleted only in the case of the status being Ready to Apply. If the status is Scheduled or Running, the havoc cannot be deleted.

The user gets detailed information about the havoc, such as category, havoc type, time mode, start time, duration, connection, tier, server, user, havoc details, status, and output.

Click the Delete Havoc button at the bottom of the window. This displays the Havoc Summary window confirming the deletion of the havoc.

ReApply Havoc

On the NetHavoc window, click ReApply Havoc menu item on the left pane. In this section, a user can reapply all the havocs that are at Completed, Stopped, or Failed status. The havoc cannot be reapplied if they are at any other status.

The user gets detailed information about the havoc, such as category, havoc type, time mode, start time, duration, connection, tier, server, user, havoc details, status, and output. All these havocs are at Completed, Stopped, or Failed status.

Click the check box against the havoc that needs to be reapplied, and then click the Proceed to Reapply Havoc button at the bottom of the window. This displays a window where the user can change only the Injection Time of that havoc.

After updating the configurations, click the Reapply havoc button at the bottom of the window to apply the changes.